Systems and methods for triggering actions based on touch-free gesture detection

ABSTRACT

Systems, methods and non-transitory computer-readable media for triggering actions based on touch-free gesture detection are disclosed. The disclosed systems may include at least one processor. A processor may be configured to receive image information from an image sensor, detect in the image information a gesture performed by a user, detect a location of the gesture in the image information, access information associated with at least one control boundary, the control boundary relating to a physical dimension of a device in a field of view of the user, or a physical dimension of a body of the user as perceived by the image sensor, and cause an action associated with the detected gesture, the detected gesture location, and a relationship between the detected gesture location and the control boundary.

This application is a continuation of U.S. application Ser. No.14/078,636, filed on Nov. 13, 2013, which claims the priority benefit ofU.S. provisional Application No. 61/725,559, filed on Dec. 13, 2012,both of which are incorporated herein by reference.

TECHNICAL FIELD

The present disclosure relates to the field of touch-free gesturedetection and, more particularly, systems and computer-readable mediafor causing an action to occur based on a detected touch-free gestureusing a control boundary.

BACKGROUND

Permitting a user to interact with a device or an application running ona device is useful in many different settings. For example, keyboards,mice, and joysticks are often included with electronic systems to enablea user to input data, manipulate data, and cause a processor of thesystem to cause a variety of other actions. Increasingly, however,touch-based input devices, such as keyboards, mice, and joysticks, arebeing replaced by, or supplemented with, devices that permit touch-freeuser interaction. For example, a system may include an image sensor tocapture images of a user, including, for example, a user's hands and/orfingers. A processor may be configured to receive such images and causeactions to occur based on touch-free gestures performed by the user.

It may be desirable to permit a user to make a number of differenttouch-free gestures that can be recognized by a system. However, thenumber of different types of touch-free gestures that can be detectedand acted upon by a system is often limited. Improvements in techniquesfor detecting and acting upon touch-free gestures are desirable.

SUMMARY

In one disclosed embodiment, a touch-free gesture recognition system isdescribed. The touch-free gesture recognition system may include atleast one processor configured to receive image information from animage sensor, detect in the image information a gesture performed by auser, detect a location of the gesture in the image information, accessinformation associated with at least one control boundary, the controlboundary relating to a physical dimension of a device in a field of viewof the user, or a physical dimension of a body of the user as perceivedby the image sensor and cause an action associated with the detectedgesture, the detected gesture location, and a relationship between thedetected gesture location and the control boundary.

In another disclosed embodiment, a non-transitory computer-readablemedium is described. The non-transitory computer-readable medium mayinclude instructions that, when executed by a processor, cause theprocessor to perform operations. The operations include receiving imageinformation from an image sensor, detecting in the image information agesture performed by a user, detecting a location of the gesture in theimage information, accessing information associated with at least onecontrol boundary, the control boundary relating to a physical dimensionof a device in a field of view of the user, or a physical dimension of abody of the user as perceived by the image sensor, and causing an actionassociated with the detected gesture, the detected gesture location, anda relationship between the detected gesture location and the controlboundary.

Additional aspects related to the embodiments will be set forth in partin the description which follows, and in part will be understood fromthe description, or may be learned by practice of the invention.

It is to be understood that both the foregoing general description andthe following detailed description are exemplary and explanatory onlyand are not restrictive of the invention, as claimed.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 illustrates an example touch-free gesture recognition system thatmay be used for implementing the disclosed embodiments.

FIG. 2 illustrates example operations that a processor of a touch-freegesture recognition system may be configured to perform, in accordancewith some of the disclosed embodiments.

FIG. 3 illustrates an example implementation of a touch-free gesturerecognition system in accordance with some of the disclosed embodiments.

FIG. 4 illustrates another example implementation of a touch-freegesture recognition system in accordance with some of the disclosedembodiments.

FIGS. 5A-5L illustrate graphical representations of example motion pathsthat may be associated with touch-free gesture systems and methodsconsistent with the disclosed embodiments.

FIG. 6 illustrates a few exemplary hand poses that may be associatedwith touch-free gesture systems and methods consistent with thedisclosed embodiments.

DETAILED DESCRIPTION

Reference will now be made in detail to the example embodiments, whichare illustrated in the accompanying drawings. Wherever possible, thesame reference numbers will be used throughout the drawings to refer tothe same or like parts.

A touch-free gesture recognition system is disclosed. A touch-freegesture recognition system may be any system in which, at least at somepoint during user interaction, the user is able to interact withoutphysically contacting an interface such as, for example, a keyboard,mouse, or joystick. In some embodiments, the system includes at leastone processor configured to receive image information from an imagesensor. The processor may be configured to detect in the imageinformation of a gesture performed by the user (e.g., a hand gesture)and to detect a location of the gesture in the image information.Moreover, in some embodiments, the processor is configured to accessinformation associated with at least one control boundary, the controlboundary relating to a physical dimension of a device in a field of viewof the user, or a physical dimension of a body of the user as perceivedby the image sensor. For example, and as described later in greaterdetail, a control boundary may be representative of an orthogonalprojection of the physical edges of a device (e.g., a display) into 3Dspace or a projection of the physical edges of the device as is expectedto be perceived by the user. Alternatively, or additionally, a controlboundary may be representative of, for example, a boundary associatedwith the user's body (e.g., a contour of at least a portion of a user'sbody or a bounding shape such as a rectangular-shape surrounding acontour of a portion of the user's body). As described later in greaterdetail, a body of the user as perceived by the image sensor includes,for example, any portion of the image information captured by the imagesensor that is associated with the visual appearance of the user's body.

In some embodiments, the processor is configured to cause an actionassociated with the detected gesture, the detected gesture location, anda relationship between the detected gesture location and the controlboundary. The action performed by the processor may be, for example,generation of a message or execution of a command associated with thegesture. For example, the generated message or command may be addressedto any type of destination including, but not limited to, an operatingsystem, one or more services, one or more applications, one or moredevices, one or more remote applications, one or more remote services,or one or more remote devices.

For example, the action performed by the processor may comprisecommunicating with an external device or website responsive to selectionof a graphical element. For example, the communication may includesending a message to an application running on the external device, aservice running on the external device, an operating system running onthe external device, a process running on the external device, one ormore applications running on a processor of the external device, asoftware program running in the background of the external device, or toone or more services running on the external device. Moreover, forexample, the action may include sending a message to an applicationrunning on a device, a service running on the device, an operatingsystem running on the device, a process running on the device, one ormore applications running on a processor of the device, a softwareprogram running in the background of the device, or to one or moreservices running on the device.

The action may also include, for example, responsive to a selection of agraphical element, sending a message requesting data relating to agraphical element identified in an image from an application running onthe external device, a service running on the external device, anoperating system running on the external device, a process running onthe external device, one or more applications running on a processor ofthe external device, a software program running in the background of theexternal device, or to one or more services running on the externaldevice. The action may also include, for example, responsive to aselection of a graphical element, sending a message requesting a datarelating to a graphical element identified in an image from anapplication running on a device, a service running on the device, anoperating system running on the device, a process running on the device,one or more applications running on a processor of the device, asoftware program running in the background of the device, or to one ormore services running on the device.

The action may also include a command selected, for example, from acommand to run an application on the external device or website, acommand to stop an application running on the external device orwebsite, a command to activate a service running on the external deviceor website, a command to stop a service running on the external deviceor website, or a command to send data relating to a graphical elementidentified in an image.

A message to a device may be a command. The command may be selected, forexample, from a command to run an application on the device, a commandto stop an application running on the device or website, a command toactivate a service running on the device, a command to stop a servicerunning on the device, or a command to send data relating to a graphicalelement identified in an image.

The action may also include, for example responsive to a selection of agraphical element, receiving from the external device or website datarelating to a graphical element identified in an image and presentingthe received data to a user. The communication with the external deviceor website may be over a communication network.

Gestures may be one-handed or two handed. Exemplary actions associatedwith a two-handed gesture can include, for example, selecting an area,zooming in or out of the selected area by moving the fingertips awayfrom or towards each other, rotation of the selected area by arotational movement of the fingertips. Actions associated with atwo-finger pointing gesture can include creating an interaction betweentwo objects, such as combining a music track with a video track or for agaming interaction such as selecting an object by pointing with onefinger, and setting the direction of its movement by pointing to alocation on the display with another finger.

FIG. 1 is a diagram illustrating an example touch-free gesturerecognition system 100 that may be used for implementing the disclosedembodiments. System 100 may include, among other things, one or moredevices 2, illustrated generically in FIG. 1. Device 2 may be, forexample, a personal computer (PC), an entertainment device, a set topbox, a television, a mobile game machine, a mobile phone, a tabletcomputer, an e-reader, a portable game console, a portable computer suchas a laptop or ultrabook, a home appliance such as a kitchen appliance,a communication device, an air conditioning thermostat, a dockingstation, a game machine such as a mobile video gaming device, a digitalcamera, a watch, an entertainment device, speakers, a Smart Home device,a media player or media system, a location-based device, a picoprojector or an embedded projector, a medical device such as a medicaldisplay device, a vehicle, an in-car/in-air infotainment system, anavigation system, a wearable device, an augmented reality-enableddevice, wearable goggles, a robot, interactive digital signage, adigital kiosk, a vending machine, an automated teller machine (ATM), orany other apparatus that may receive data from a user or output data toa user. Moreover, device 2 may be handheld (e.g., held by a user's hand19) or non-handheld.

System 100 may include some or all of the following components: adisplay 4, image sensor 6, keypad 8 comprising one or more keys 10,processor 12, memory device 16, and housing 14. In some embodiments,some or all of the display 4, image sensor 6, keypad 8 comprising one ormore keys 10, processor 12, housing 14, and memory device 16, arecomponents of device 2. However, in some embodiments, some or all of thedisplay 4, image sensor 6, keypad 8 comprising one or more keys 10,processor 12, housing 14, and memory device 16, are separate from, butconnected to the device 2 (using either a wired or wireless connection).For example, image sensor 6 may be located apart from device 2.Moreover, in some embodiments, components such as, for example, thedisplay 4, keypad 8 comprising one or more keys 10, or housing 14, areomitted from system 100.

A display 4 may include, for example, one or more of a television set,computer monitor, head-mounted display, broadcast reference monitor, aliquid crystal display (LCD) screen, a light-emitting diode (LED) baseddisplay, an LED-backlit LCD display, a cathode ray tube (CRT) display,an electroluminescent (ELD) display, an electronic paper/ink display, aplasma display panel, an organic light-emitting diode (OLED) display,thin-film transistor display (TFT), High-Performance Addressing display(HPA), a surface-conduction electron-emitter display, a quantum dotdisplay, an interferometric modulator display, a swept-volume display, acarbon nanotube display, a variforcal mirror display, an emissive volumedisplay, a laser display, a holographic display, a light field display,a projector and surface upon which images are projected, or any otherelectronic device for outputting visual information. In someembodiments, the display 4 is positioned in the touch-free gesturerecognition system 100 such that the display 4 is viewable by one ormore users.

Image sensor 6 may include, for example, a CCD image sensor, a CMOSimage sensor, a camera, a light sensor, an IR sensor, an ultrasonicsensor, a proximity sensor, a shortwave infrared (SWIR) image sensor, areflectivity sensor, or any other device that is capable of sensingvisual characteristics of an environment. Moreover, image sensor 6 mayinclude, for example, a single photosensor or 1-D line sensor capable ofscanning an area, a 2-D sensor, or a stereoscopic sensor that includes,for example, a plurality of 2-D image sensors. Image sensor 6 may beassociated with a lens for focusing a particular area of light onto theimage sensor 6. In some embodiments, image sensor 6 is positioned tocapture images of an area associated with at least some display-viewablelocations. For example, image sensor 6 may be positioned to captureimages of one or more users viewing the display 4. However, a display 4is not necessarily a part of system 100, and image sensor 6 may bepositioned at any location to capture images of a user and/or of device2.

Image sensor 6 may view, for example, a conical or pyramidal volume ofspace 18, as indicated by the broken lines in FIG. 1. The image sensor 6may have a fixed position on the device 2, in which case the viewingspace 18 is fixed relative to the device 2, or may be positionablyattached to the device 2 or elsewhere, in which case the viewing space18 may be selectable. Images captured by the image sensor 6 may bedigitized by the image sensor 6 and input to the processor 12, or may beinput to the processor 12 in analog form and digitized by the processor12.

Some embodiments may include at least one processor. The at least oneprocessor may include any electric circuit that may be configured toperform a logic operation on at least one input variable, including, forexample one or more integrated circuits, microchips, microcontrollers,and microprocessors, which may be all or part of a central processingunit (CPU), a digital signal processor (DSP), a field programmable gatearray (FPGA), a graphical processing unit (GPU), or any other circuitknown to those skilled in the art that may be suitable for executinginstructions or performing logic operations. Multiple functions may beaccomplished using a single processor or multiple related and/orunrelated functions may be divide among multiple processors.

In some embodiments, such is illustrated in FIG. 1, at least oneprocessor may include processor 12 connected to memory 16. Memory 16 mayinclude, for example, persistent memory, ROM, EEPROM, EAROM, flashmemory devices, magnetic disks, magneto optical disks, CD-ROM, DVD-ROM,Blu-ray, and the like, and may contain instructions (i.e., software orfirmware) or other data. Generally, processor 12 may receiveinstructions and data stored by memory 16. Thus, in some embodiments,processor 12 executes the software or firmware to perform functions byoperating on input data and generating output. However, processor 12 mayalso be, for example, dedicated hardware or an application-specificintegrated circuit (ASIC) that performs processes by operating on inputdata and generating output. Processor 12 may be any combination ofdedicated hardware, one or more ASICs, one or more general purposeprocessors, one or more DSPs, one or more GPUs, or one or more otherprocessors capable of processing digital information.

FIG. 2 illustrates exemplary operations 200 that at least one processormay be configured to perform. For example, as discussed above, processor12 of the touch-free gesture recognition system 100 may be configured toperform these operations by executing software or firmware stored inmemory 16, or may be configured to perform these operations usingdedicated hardware or one or more ASICs.

In some embodiments, at least one processor may be configured to receiveimage information from an image sensor (operation 210).

In order to reduce data transfer from the image sensor 6 to an embeddeddevice motherboard, general purpose processor, application processor,GPU a processor controlled by the application processor, or any otherprocessor, including, for example, processor 12, the gesture recognitionsystem may be partially or completely be integrated into the imagesensor 6. In the case where only partial integration to the imagesensor, ISP or image sensor module takes place, image preprocessing,which extracts an object's features related to the predefined object,may be integrated as part of the image sensor, ISP or image sensormodule. A mathematical representation of the video/image and/or theobject's features may be transferred for further processing on anexternal CPU via dedicated wire connection or bus. In the case that thewhole system is integrated into the image sensor, ISP or image sensormodule, only a message or command (including, for example, the messagesand commands discussed in more detail above and below) may be sent to anexternal CPU. Moreover, in some embodiments, if the system incorporatesa stereoscopic image sensor, a depth map of the environment may becreated by image preprocessing of the video/image in each one of the 2Dimage sensors or image sensor ISPs and the mathematical representationof the video/image, object's features, and/or other reduced informationmay be further processed in an external CPU.

“Image information,” as used in this application, may be one or more ofan analog image captured by image sensor 6, a digital image captured ordetermined by image sensor 6, subset of the digital or analog imagecaptured by image sensor 6, digital information further processed by anISP, a mathematical representation or transformation of informationassociated with data sensed by image sensor 6, frequencies in the imagecaptured by image sensor 6, conceptual information such as presence ofobjects in the field of view of the image sensor 6, informationindicative of the state of the image sensor or its parameters whencapturing an image (e.g., exposure, frame rate, resolution of the image,color bit resolution, depth resolution, or field of view of the imagesensor), information from other sensors when the image sensor 6 iscapturing an image (e.g. proximity sensor information, or accelerometerinformation), information describing further processing that took placeafter an image was captured, illumination conditions when an image iscaptured, features extracted from a digital image by image sensor 6, orany other information associated with data sensed by image sensor 6.Moreover, “image information” may include information associated withstatic images, motion images (i.e., video), or any other visual-baseddata.

In some embodiments, the at least one processor may be configured todetect in the image information a gesture performed by a user (operation220). Moreover, in some embodiments, the at least one processor may beconfigured to detect a location of the gesture in the image information(operation 230). The gesture may be, for example, a gesture performed bythe user using predefined object 24 in the viewing space 16. Thepredefined object 24 may be, for example, one or more hands, one or morefingers, one or more fingertips, one or more other parts of a hand, orone or more hand-held objects associated with a user. In someembodiments, detection of the gesture is initiated based on detection ofa hand at a predefined location or in a predefined pose. For example,detection of a gesture may be initiated if a hand is in a predefinedpose and in a predefined location with respect to a control boundary.More particularly, for example, detection of a gesture may be initiatedif a hand is in an open-handed pose (e.g., all fingers of the hand awayfrom the palm of the hand) or in a first pose (e.g., all fingers of thehand folded over the palm of the hand). Detection of a gesture may alsobe initiated if, for example, a hand is detected in a predefined posewhile the hand is outside of the control boundary (e.g., for apredefined amount of time), or a predefined gesture is performed inrelation to the control boundary, Moreover, for example, detection of agesture may be initiated based on the user location, as captured byimage sensor 6 or other sensors. Moreover, for example, detection of agesture may be initiated based on a detection of another gesture. E.g.,to detect a “left to right” gesture, the processor may first detect a“waving” gesture.

As used in this application, the term “gesture” may refer to, forexample, a swiping gesture associated with an object presented on adisplay, a pinching gesture of two fingers, a pointing gesture towardsan object presented on a display, a left-to-right gesture, aright-to-left gesture, an upwards gesture, a downwards gesture, apushing gesture, a waving gesture, a clapping gesture, a reverseclapping gesture, a gesture of splaying fingers on a hand, a reversegesture of splaying fingers on a hand, a holding gesture associated withan object presented on a display for a predetermined amount of time, aclicking gesture associated with an object presented on a display, adouble clicking gesture, a right clicking gesture, a left clickinggesture, a bottom clicking gesture, a top clicking gesture, a graspinggesture, a gesture towards an object presented on a display from a rightside, a gesture towards an object presented on a display from a leftside, a gesture passing through an object presented on a display, ablast gesture, a tipping gesture, a clockwise or counterclockwisetwo-finger grasping gesture over an object presented on a display, aclick-drag-release gesture, a gesture sliding an icon such as a volumebar, or any other motion associated with a hand or handheld object. Agesture may be detected in the image information if the processor 12determines that a particular gesture has been or is being performed bythe user.

An object associated with the user may be detected in the imageinformation based on, for example, the contour and/or location of anobject in the image information. For example, processor 12 may access afilter mask associated with predefined object 24 and apply the filtermask to the image information to determine if the object is present inthe image information. That is, for example, the location in the imageinformation most correlated to the filter mask may be determined as thelocation of the object associated with predefined object 24. Processor12 may be configured, for example, to detect a gesture based on a singlelocation or based on a plurality of locations over time. Processor 12may also be configured to access a plurality of different filter masksassociated with a plurality of different hand poses. Thus, for example,a filter mask from the plurality of different filter masks that has abest correlation to the image information may cause a determination thatthe hand pose associated with the filter mask is the hand pose of thepredefined object 24. Processor 12 may be configured, for example, todetect a gesture based on a single pose or based on a plurality of posesover time. Moreover, processor 12 may be configured, for example, todetect a gesture based on both the determined one or more locations andthe determined one or more poses. Other techniques for detectingreal-world objects in image information (e.g., edge matching, greyscalematching, gradient matching, and other image feature based methods) arewell known in the art, and may also be used to detect a gesture in theimage information. For example, U.S. Patent Application Publication No.2012/0092304 and U.S. Patent Application Publication No. 2011/0291925disclose techniques for performing object detection, both of which areincorporated by reference in their entirety. Each of the above-mentionedgestures may be associated with a control boundary.

A gesture location, as used herein, may refer to one or a plurality oflocations associated with a gesture. For example, a gesture location maybe a location of an object or gesture in the image information ascaptured by the image sensor, a location of an object or gesture in theimage information in relation to one or more control boundaries, alocation of an object or gesture in the 3D space in front of the user, alocation of an object or gesture in relation to a device or physicaldimension of a device, or a location of an object or gesture in relationto the user body or part of the user body such as the user's head. Forexample, a “gesture location” may include a set of locations comprisingone or more of a starting location of a gesture, intermediate locationsof a gesture, and an ending location of a gesture. A processor 12 maydetect a location of the gesture in the image information by determininglocations on display 4 associated with the gesture or locations in theimage information captured by image sensor 6 that are associated withthe gesture (e.g., locations in the image information in which thepredefined object 24 appears while the gesture is performed). Forexample, as discussed above, processor 12 may be configured to apply afilter mask to the image information to detect an object associated withpredefined object 24. In some embodiments, the location of the objectassociated with predefined object 24 in the image information may beused as the detected location of the gesture in the image information.

In other embodiments, the location of the object associated withpredefined object 24 in the image information may be used to determine acorresponding location on display 4 (including, for example, a virtuallocation on display 4 that is outside the boundaries of display 4), andthe corresponding location on display 4 may be used as the detectedlocation of the gesture in the image information. For example, thegesture may be used to control movement of a cursor, and a gestureassociated with a control boundary may be initiated when the cursor isbrought to an edge or corner of the control boundary. Thus, for example,a user may extend a finger in front of the device, and the processor mayrecognize the fingertip, enabling the user to control a cursor. The usermay then move the fingertip to the right, for example, until the cursorreaches the right edge of the display. When the cursor reaches the rightedge of the display, a visual indication may be displayed indicating tothe user that a gesture associated with the right edge is enabled. Whenthe user then performs a gesture to the left, the gesture detected bythe processor may be associated with the right edge of the device.

The following are examples of gestures associated with a controlboundary:

-   -   “Hand-right motion”—the predefined object 24 may move from right        to left, from a location which is beyond a right edge of a        control boundary, over the right edge, to a location which is to        the left of the right edge.    -   “Hand-left motion”—the predefined object 24 may move from left        to right, from a location which is beyond a left edge of a        control boundary, over the left edge, to a location which is to        the right of the left edge.    -   “Hand-up motion”—the predefined object 24 may move upwards from        a location which is below a bottom edge of a control boundary,        over the bottom edge, to a location which is above the bottom        edge.    -   “Hand-down motion”—the predefined object 24 may move downwards        from a location which is above a top edge of a control boundary,        over the top edge, to a location which is below the top edge.    -   “Hand-corner up-right”—the predefined object 24 may begin at a        location beyond the upper-right corner of the control boundary        and move over the upper-right corner to the other side of the        control boundary.    -   “Hand-corner up-left”—the predefined object 24 may begin at a        location beyond the upper-left corner of the control boundary        and move over the upper-left corner to the other side of the        control boundary.    -   “Hand-corner down-right”—the predefined object 24 may begin at a        location beyond the lower-right corner of the control boundary        and move over the lower-right corner to the other side of the        control boundary.    -   “Hand-corner down-left”—the predefined object 24 may begin at a        location beyond the lower-left corner of the control boundary        and move over the lower-left corner to the other side of the        control boundary.

FIGS. 5A-5L depict graphical representations of a few exemplary motionpaths (e.g., the illustrated arrows) of gestures, and the gestures'relationship to a control boundary (e.g., the illustrated rectangles).FIG. 6 depicts a few exemplary representations of hand poses that may beused during a gesture, and may affect a type of gesture that is detectedand/or action that is caused by a processor. Each differing combinationof motion path and gesture may result in a differing action.

In some embodiments, the at least one processor is also configured toaccess information associated with at least one control boundary, thecontrol boundary relating to a physical dimension of a device in a fieldof view of the user, or a physical dimension of a body of the user asperceived by the image sensor (operation 240). In some embodiments theprocessor 12 is configured to generate the information associated withthe control boundary prior to accessing the information. However, theinformation may also, for example, be generated by another device,stored in memory 16, and accessed by processor 12. Accessing informationassociated with at least one control boundary may include any operationperformed by processor 12 in which the information associated with theleast one control boundary is acquired by processor 12. For example, theinformation associated with at least one control boundary may bereceived by processor 12 from memory 16, may be received by processor 12from an external device, or may be determined by processor 12.

A control boundary may be determined (e.g., by processor 12 or byanother device) in a number of different ways. As discussed above, acontrol boundary may relate to one or more of a physical dimension of adevice, which may, for example, be in a field of view of the user, aphysical location of the device, the physical location of the device inrelate to the location of the user, physical dimensions of a body asperceived by the image sensor, or a physical location of a user's bodyor body parts as perceived by the image sensor. A control boundary maybe determined from a combination of information related to physicaldevices located in the physical space where the user performs a gestureand information related to the physical dimensions of the user's body inthat the physical space. Moreover, a control boundary may relate to partof a physical device, and location of such part. For example, thelocation of speakers of a device may be used to determine a controlboundary (e.g., the edges and corners of a speaker device), so that if auser performs gestures associated with the control boundary (e.g., adownward gesture along or near the right edge of the control boundary,as depicted, for example, in FIG. 5L), the volume of the speakers may becontrolled by the gesture. A control boundary may also relate one ormore of a specific location on the device, such as the location of themanufacturer logo, or components on the device. Furthermore, the controlboundary may also relate to virtual objects as perceived by the user.Virtual objects may be objects displayed to the user in 3D space in theuser's field of view by a 3D display device or by a wearable displaydevice, such as wearable augmented reality glasses. Virtual objects, forexample, may include icons, images, video, or any kind of visualinformation that can be perceived by the user in real or virtual 3D. Asused in this application, a physical dimension of a device may include adimension of a virtual object.

FIG. 3 depicts an exemplary implementation of a touch-free gesturerecognition system in accordance with some embodiments in which thecontrol boundary may relate to a physical dimension of a device in afield of view of the user. FIG. 4 depicts an exemplary implementation ofa touch-free gesture recognition system in accordance with someembodiments in which the control boundary may relate to a physicaldimension of a body of the user.

As depicted in the example implementation in FIG. 3, user 30 may viewdisplay 4 within the conical or pyramidal volume of space 18 viewable byimage sensor 6. In some embodiments, the control boundary relates tobroken lines AB and CD, which extend perpendicularly from definedlocations on the device, such as, for example, the left and right edgesof display 4. For example, as discussed below, the processor 12 may beconfigured to determine one or more locations in the image informationthat correspond to lines AB and CD. While only broken lines AB and CDare depicted in FIG. 3, associated with the left and right edges ofdisplay 4, in some embodiments the control boundary may additionally oralternatively be associated with the top and bottom edges of display 4,or some other physical dimension of the display, such as a border,bevel, or frame of the display, or a reference presented on the display.Moreover, while the control boundary may be determined based on thephysical dimensions or other aspects of display 4, the control boundarymay also be determined based on the physical dimensions of any otherdevice (e.g., the boundaries or contour of a stationary object).

The processor 12 may be configured to determine the location anddistance of the user from the display 4. For example, the processor 12may use information from a proximity sensor, a depth sensing sensor,information representative of a 3D map in front of the device, or useface detection to determine the location and distance of the user fromthe display 4, and from the location and distance compute a field ofview (FOV) of the user. For example, an inter-pupillary distance in theimage information may be measured and used to determine the location anddistance of the user from the display 4. For example, the processor maybe configured to compare the inter-pupillary distance in the imageinformation to a known or determined inter-pupillary distance associatedwith the user, and determine a distance based on the difference (as theuser stands further from image sensor 6, the inter-pupillary distance inthe image information may decrease). The accuracy of the user distancedetermination may be improved by utilizing the user's age, since, forexample, a younger user may have a smaller inter-pupillary distance.Face recognition may also be applied to identify the user and retrieveinformation related to the identified user. For example, an Internetsocial medium (e.g., Facebook) may be accessed to obtain informationabout the user (e.g., age, pictures, interests, etc.). This informationmay be used to improve the accuracy of the inter-pupillary distance, andthus improve the accuracy of the distance calculation of the user fromthe screen.

The processor 12 may also be configured to determine an average distancedz in front of the user's eyes that the user positions the predefinedobject 24 when performing a gesture. The average distance dz may dependon the physical dimensions of the user (e.g., the length of the user'sforearm), which can be estimated, for example, from the user'sinter-pupillary distance. A range of distances (e.g., dz+Δz throughdz-Δz) surrounding the average distance dz may also be determined.During the performance of a gesture, the predefined object 24 may oftenbe found at a distance in the interval between dz+Δz to dz-Δz. In someembodiments, Δz may be predefined. Alternatively, Δz may be calculatedas a fixed fraction (e.g., 0.2) of dz. As depicted in FIG. 3, brokenline FJ substantially parallel to the display 4 at a distance dz-Δz fromthe user may intersect the broken lines AB and CD at points F and J.Points F and J may be representative of a region of the viewing space ofthe image sensor 6 having semi-apical angle α, indicated by the brokenlines GJ and GF, which serve to determine the control boundary. Thus,for example, if the user's hand 32 is outside of the region bounded bythe lines GJ and GF, the hand 32 may be considered to be outside thecontrol boundary. Thus, in some embodiments, the information associatedwith the control boundary may be, for example, the locations of lines GJand GF in the image information, or information from which the locationsof lines GJ and GF in the image information can be determined.

Alternatively or additionally, in some embodiments, at least oneprocessor is configured to determine the control boundary based, atleast in part, on a dimension of the device (e.g., display 4) as isexpected to be perceived by the user. For example, broken lines BE andBD in FIG. 3, which extend from a location on or near the body of theuser (determined, for example, based on the distance from the imagesensor 6 to the user, the location of the user's face or eyes, and/orthe FOV of the user) to the left and right edges of display 4, arerepresentative of dimensions of display 4 as is expected to be perceivedby the user. That is, based on the distance and orientation of the userrelative to the display 4, the processor may be configured to determinehow the display is likely perceived from the vantage point of the user.(E.g., by determining sight lines from the user to the edges of thedisplay.) Thus, the processor may be configured to determine the controlboundary by determining one or more locations in the image informationthat correspond to lines BE and BD (e.g., based on an analysis of theaverage distance from the user's body that the user positions thepredefined object 24). While only broken lines BE and BD are depicted inFIG. 3, associated with the left and right edges of display 4, in someembodiments the control boundary may additionally or alternatively beassociated with the top and bottom edges of display 4.

Alternatively or additionally, the control boundary may relate to aphysical dimension of a body of the user as perceived by the imagesensor. That is, based on the distance and/or orientation of the userrelative to the display or image sensor, the processor may be configuredto determine a control boundary. The farther the user from the display,the smaller the image sensor's perception of the user, and the smalleran area bounded by the control boundaries. The processor may beconfigured to identify specific portions of a user's body for purposesof control boundary determination. Thus the control boundary may relateto the physical dimensions of the user's torso, shoulders, head, hand,or any other portion or portions of the user's body. The controlboundary may be related to the physical dimension of a body portion byeither relying on the actual or approximate dimension of the bodyportion, or by otherwise using the body portion as a reference forsetting control boundaries. (E.g., a control boundary may be set apredetermined distance from a reference location on the body portion.)

The processor 12 may be configured to determine a contour of a portionof a body of the user (e.g., a torso of the user) in the imageinformation received from image sensor 6. Moreover, the processor 12 maybe configured to determine, for example, an area bounding the user(e.g., a bounding box surrounding the entire user or the torso of theuser). For example, the broken lines KL and MN depicted in FIG. 4 areassociated with the left and right sides of a contour or area boundingthe user. The processor 12 may be configured to determine the controlboundary by determining one or more locations in the image informationthat correspond to the determined contour or bounding area. Thus, forexample, the processor 12 may be configured to determine the controlboundary by detecting a portion of a body of the user, other than theuser's hand (e.g., a torso), and to define the control boundary based onthe detected body portion. While only broken lines associated with theleft and right sides of the user are depicted in FIG. 4, in someembodiments the control boundary may additionally or alternatively beassociated with the top and bottom of the contour or bounding area.

In some embodiments, the at least on processor may be configured tocause a visual or audio indication when the control boundary is crossed.For example, if an object in the image information associated withpredefined object 24 crosses the control boundary, this indication mayinform the user that a gesture performed within a predefined amount oftime will be interpreted as gesture associated with the controlboundary. For example, if an edge of the control boundary is crossed, anicon may begin to fade-in on display 4. If the gesture is completedwithin the predefined amount of time, the icon may be finalized; if thegesture is not completed within the predefined amount of time, the iconmay no longer be presented on display 4.

While a control boundary is discussed above with respect to a singleuser, the same control boundary may be associated with a plurality ofusers. For example, when a gesture performed by one user is detected, acontrol boundary may be accessed that was determined for another user,or that was determined for a plurality of users. Moreover, the controlboundary may be determined based on an estimated location of a user,without actually determining the location of the user.

In some embodiments, the at least one processor is also configured tocause an action associated with the detected gesture, the detectedgesture location, and a relationship between the detected gesturelocation and the control boundary (operation 250). As discussed above,an action caused by a processor may be, for example, generation of amessage or execution of a command associated with the gesture. A messageor command may be, for example, addressed to one or more operatingsystems, one or more services, one or more applications, one or moredevices, one or more remote applications, one or more remote services,or one or more remote devices. In some embodiments, the action includesan output to a user. For example, the action may provide an indicationto a user that some event has occurred. The indication may be, forexample, visual (e.g., using display 4), audio, tactile, ultrasonic, orhaptic. An indication may be, for example, an icon presented on adisplay, change of an icon presented on a display, a change in color ofan icon presented on a display, an indication light, an indicator movingon a display, a directional vibration indication, or an air tactileindication. Moreover, for example, the indicator may appear on top ofall other images appearing on the display.

In some embodiments, memory 16 stores data (e.g., a look-up table) thatprovides, for one or more predefined gestures and/or gesture locations,one or more corresponding actions to be performed by the processor 12.Each gesture that is associated with a control boundary may becharacterized by one or more of the following factors: the startingpoint of the gesture, the motion path of the gesture (e.g., asemicircular movement, a back and forth movement, an “S”-like path, or atriangular movement), the specific edges or corners of the controlboundary crossed by the path, the number of times an edge or corner ofthe control boundary is crossed by the path, and where the path crossesedges or corners of the control boundary. By way of example only, agesture associated with a right edge of a control boundary may toggle acharm menu, a gesture associated with a top edge of a control boundaryor bottom edge of a control boundary may toggle an application command,a gesture associated with a left edge of a control boundary may switchto a last application, and a gesture associated with both a right edgeand a left edge of a control boundary (e.g., as depicted in FIG. 5K) mayselect an application or start menu. As an additional example, if agesture crosses a right edge of a control boundary, an image of avirtual page may progressively cross leftward over the right edge of thedisplay so that the virtual page is progressively displayed on thedisplay; the more the predefined object associated with the user ismoved away from the right edge of the screen, the more the virtual pageis displayed on the screen.

For example, processor 12 may be configured to cause a first action whenthe gesture is detected crossing the control boundary, and to cause asecond action when the gesture is detected within the control boundary.That is, the same gesture may result in a different action based onwhether the gesture crosses the control boundary. For example, a usermay perform a right-to-left gesture. If the right-to-left gesture isdetected entirely within the control boundary, the processor may beconfigured, for example, to shift a portion of the image presented ondisplay 4 to the left (e.g., a user may use the right-to-left gesture tomove a photograph presented on display 4 in a leftward direction). If,however, the right-to-left gesture is detected to cross the right edgeof the control boundary, the processor may be configured, by way ofexample only, to replace the image presented on display 4 with anotherimage (e.g., a user may use the right-to-left gesture to scroll throughphotographs in a photo album).

Moreover, for example, the processor 12 may be configured to distinguishbetween a plurality of predefined gestures to cause a plurality ofactions, each associated with a differing predefined gesture. Forexample, if differing hand poses cross the control boundary at the samelocation, the processor may cause differing actions. For example, apointing finger crossing the control boundary may cause a first action,while an open hand crossing the control boundary may cause a differingsecond action. As an alternative example, if a user performs aright-to-left gesture that is detected to cross the right edge of thecontrol boundary, the processor may cause a first action, but crossingthe control boundary in the same location with the same hand pose, butfrom a different direction, may cause a second action. As anotherexample, a gesture performed in a first speed may cause a first action;the same gesture, when performed in second speed, may cause a secondaction. As another example, a left-to-right gesture performed in a firstmotion path representative of the predefined object (e.g., the user'shand) moving a first distance (e.g. 10 cm) may cause a first action; thesame gesture performed in a second motion path representative of thepredefined object moving a second distance (e.g. 30 cm) may cause asecond action The first and second actions could be any message orcommand. By way of example only, the first action may replace the imagepresented on display 4 with a previously viewed image, while the secondaction may cause a new image to be displayed.

Moreover, for example, the processor 12 may be configured to generate aplurality of actions, each associated with a differing relative positionof the gesture location to the control boundary. For example, if a firstgesture (e.g. left to right gesture) crosses a control boundary near thecontrol boundary top, the processor may be configured to generate afirst action, while if the same first gesture, crosses the controlboundary near the control boundary bottom, the processor may beconfigured to generate a second action. Another example, if a gesturethat crosses the control boundary begins at a location outside of thecontrol boundary by more than a predetermined distance, the processormay be configured to generate a first action. However, if a gesture thatcrosses the control boundary begins at a location outside of the controlboundary by less than a predetermined distance, the processor may beconfigured to generate a second action. By way of example only, thefirst action may cause an application to shut down while the secondaction may close a window of the application.

Moreover, for example, the action may be associated with a predefinedmotion path associated with the gesture location and the controlboundary. For example, memory 16 may store a plurality of differingmotion paths, with each detected path causing a differing action. Apredefined motion path may include a set of directions of a gesture(e.g., left, right, up down, left-up, left-down, right-up, orright-down) in a chronological sequence. Or, a predefined motion pathmay be one that crosses multiple boundaries (e.g., slicing a corner orslicing across entire display), or one that crosses a boundary in aspecific region (e.g., crosses top right).

A predefined motion path may also include motions associated with aboundary, but which do not necessarily cross a boundary. (E.g., up downmotion outside right boundary; up down motion within right boundary).

Moreover, a predefined motion path may be defined by a series of motionsthat change direction in a specific chronological sequence. (E.g., afirst action may be caused by down-up, left right; while a second actionmay be caused by up-down, left-right).

Moreover, a predefined motion path may be defined by one or more of thestarting point of the gesture, the motion path of the gesture (e.g., asemicircular movement, a back and forth movement, an “S”-like path, or atriangular movement), the specific edges or corners of the controlboundary crossed by the path, the number of times an edge or corner ofthe control boundary is crossed by the path, and where the path crossesedges or corners of the control boundary.

In some embodiments, as discussed above, the processor may be configuredto determine the control boundary by detecting a portion of a body ofthe user, other than the user's hand (e.g., a torso), and to define thecontrol boundary based on the detected body portion. In someembodiments, the processor may further be configured to generate theaction based, at least in part, on an identity of the gesture, and arelative location of the gesture to the control boundary. Each differentpredefined gesture (e.g., hand pose) may have a differing identity.Moreover, a gesture may be performed at different relative locations tothe control boundary, enabling each different combination ofgesture/movement relative to the control boundary to cause a differingaction.

In addition, the processor 12 may be configured to perform differentactions based on the number of times a control boundary is crossed or alength of the path of the gesture relative to the physical dimensions ofthe user's body. For example, an action may be caused by the processorbased on a number of times that each edge or corner of the controlboundary is crossed by a path of a gesture. By way of another example, afirst action may be caused by the processor if a gesture, having a firstlength, is performed by a first user of a first height. The first actionmay also be caused by the processor if a gesture, having a secondlength, is performed by a second user of a second height, if the secondlength as compared to the second height is substantially the same as thefirst length as compared to the first height. In this example scenario,the processor may cause a second action if a gesture, having the secondlength, is performed by the first user.

The processor 12 may be configured to cause a variety of actions forgestures associated with a control boundary. For example, in addition tothe examples discussed above, the processor 12 may be configured toactivate a toolbar presented on display 4, which is associated with aparticular edge of the control boundary, based on the gesture location.That is, for example, if it is determined that the gesture crosses aright edge of the control boundary, a toolbar may be displayed along theright edge of display 4. Additionally, for example, the processor 12 maybe configured to cause an image to be presented on display 4 based onthe gesture, the gesture location, and the control boundary (e.g., anedge crossed by the gesture).

By configuring a processor to cause an action associated with a detectedgesture, the detected gesture location, and a relationship between thedetected gesture location and a control boundary, a more robust numberof types of touch-free gestures by a user can be performed and detected.Moreover, touch-free gestures associated with a control boundary mayincrease the usability of a device that permits touch-free gestures toinput data or control operation of the device.

Certain features which, for clarity, are described in this specificationin the context of separate embodiments, may also be provided incombination in a single embodiment. Conversely, various features which,for brevity, are described in the context of a single embodiment, mayalso be provided in multiple embodiments separately or in any suitablesub-combination. Moreover, although features may be described above asacting in certain combinations and even initially claimed as such, oneor more features from a claimed combination can in some cases be excisedfrom the combination, and the claimed combination may be directed to asubcombination or variation of a subcombination.

Particular embodiments have been described. Other embodiments are withinthe scope of the following claims.

What is claimed is:
 1. A touch-free gesture recognition system,comprising: at least one processor configured to: receive imageinformation from an image sensor; determine one or more locations in theimage information relating to at least one control boundary based, atleast in part, on a physical dimension of a body of a user; detect inthe image information a gesture performed by the user; detect a gesturelocation in the image information, wherein at least a portion of thegesture location is detected outside the control boundary, and whereinthe gesture location is associated with at least one control boundarycrossed by the detected gesture; and cause an action associated with thedetected gesture, the detected gesture location, and a relationshipbetween the detected gesture location and the control boundary.
 2. Thesystem of claim 1, wherein the processor is further configured todetermine the control boundary based, at least in part, on the physicaldimension of the body of the user as perceived by the image sensor. 3.The system of claim 1, wherein the processor is further configured todistinguish between a plurality of predefined gestures to cause aplurality of actions, each associated with a differing predefinedgesture.
 4. The system of claim 1, wherein the processor is furtherconfigured to generate a plurality of actions, each associated with adiffering relative position of the gesture location to the controlboundary.
 5. The system of claim 1, wherein the physical dimension isassociated with a contour of at least a portion of the body of the user,other than the user's hand.
 6. The system of claim 1, wherein theprocessor is further configured to determine the control boundary basedon a contour of at least a portion of a body of the user in the imageinformation.
 7. The system of claim 1, wherein the action is related toa number of times the at least one control boundary is crossed by a pathof the gesture.
 8. The system of claim 1, wherein the action isassociated with a predefined motion path associated with the gesturelocation and the at least one control boundary.
 9. The system of claim1, wherein the action is associated with a predefined motion pathassociated with the at least one control boundary crossed by the gesturelocation.
 10. The system of claim 1, wherein the processor is furtherconfigured to detect a hand in a predefined location relating to thecontrol boundary and initiate detection of the gesture based on thedetection of the hand at the predefined location.
 11. The system ofclaim 1, wherein the processor is further configured to cause at leastone of a visual or audio indication when the control boundary iscrossed.
 12. The system of claim 1, wherein prior to the gesturedetection, the at least one processor is configured to provide anindication that a gesture associated with the at least one controlboundary is enabled.
 13. The system of claim 1, wherein the detectedgesture comprises an identified predefined hand pose and an identifiedpredefined motion path.
 14. A method for a touch-free gesturerecognition system, comprising: receiving image information from animage sensor; determining one or more locations in the image informationrelating to at least one control boundary based, at least in part, on aphysical dimension of a body of a user; detecting in the imageinformation a gesture performed by the user; detecting a gesturelocation in the image information, wherein at least a portion of thegesture location is detected outside the control boundary, and whereinthe gesture location is associated with at least one control boundarycrossed by the detected gesture; and causing an action associated withthe detected gesture, the detected gesture location, and a relationshipbetween the detected gesture location and the control boundary.
 15. Themethod of claim 14, wherein the physical dimension is a physicaldimension of the body of the user as perceived by the image sensor. 16.The method of claim 14, further comprising generating a plurality ofactions, each associated with a differing relative position of thegesture location to the control boundary.
 17. The method of claim 14,wherein the physical dimension is associated with a contour of at leasta portion of the body of the user, other than the user's hand.
 18. Themethod of claim 14, wherein the action is associated with a predefinedmotion path associated with the gesture location and the at least onecontrol boundary.
 19. The method of claim 14, wherein the action isassociated with a predefined motion path associated with the at leastone control boundary crossed by the gesture location.
 20. The method ofclaim 14, further comprising detecting a hand in predefined locationrelating to the control boundary and initiating detection of the gesturebased on the detection of the hand at the predefined location.
 21. Themethod of claim 14, further comprising providing, prior to the gesturedetection, an indication that a gesture associated with the at least onecontrol boundary is enabled.
 22. The method of claim 14, furthercomprising distinguishing between a plurality of predefined motion pathsassociated with the same gesture to cause a plurality of actions, eachaction associated with a different motion path.
 23. The method of claim14, wherein detecting the gesture comprises identifying a predefinedhand pose and a predefined motion path.
 24. A touch-free gesturerecognition system, comprising: at least one processor configured to:receive image information from an image sensor; determine at least onelocation in the image information associated with a control boundarybased, at least in part, on a physical dimension of a body of a user;detect in the image information a gesture performed by the user, whereinat least a portion of the gesture is detected outside the controlboundary; determine whether the detected gesture crosses the at leastone location associated with the control boundary; cause a first actionwhen the detected gesture crosses the at least one location associatedwith the control boundary; and cause a second action when the detectedgesture does not cross the at least one location associated with thecontrol boundary.
 25. The system of claim 24, wherein the detectedgesture comprises an identified predefined hand pose and an identifiedpredefined motion path.